Image pickup apparatus, control method, and storage medium

ABSTRACT

There is provided an image pickup apparatus comprising an image sensor configured to successively pick up a plurality of images for compositing of a panorama image, a memory configured to store instructions, a monitor, and a processor connected to the memory. The processor executes the instructions to specify a region that is not to be used in compositing of the panorama image in each of the plurality of images, based on a state of an optical system used at a time of pick-up of the plurality of images. The monitor displays the specified region along with an image.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image pickup apparatus, a controlmethod, and a storage medium.

Description of the Related Art

Technology for generating a panoramic full image (panorama image) from aplurality of unit images is known (Japanese Patent Laid-Open No.2005-328497). The image pickup apparatus of Japanese Patent Laid-OpenNo. 2005-328497 successively changes a shooting direction and picks upunit images that form parts of a shooting range, extracts image regionsof a predetermined size that form parts of the unit images to createmutually overlapping regions, and successively superimposes the imageregions that have been extracted. Accordingly, a panorama image thatshows the entire shooting range is generated. Furthermore, JapanesePatent Laid-Open No. 11-331696 discloses technology in which, if it isdetermined that the method of shooting is panning shooting, thepicked-up images are subject to a cylindrical coordinate transform ontoa virtual cylindrical surface that has the focal distance as its radius,and a panorama image is generated from the transformed images.

As shown in FIG. 3A, when a cylindrical coordinate transform isperformed, the size (pixel count) in the up-down direction (heightdirection) becomes smaller than in the original image at both ends of atransformed image. Accordingly, when the camera is moved in thehorizontal direction to pick up a panorama image, processing isperformed that generates a panorama image from the central regionwithout using the end portions of the transformed image (see FIG. 3B) inorder to suppress a reduction in the pixel count in the up-downdirection of the panorama image that is eventually generated. In FIG.3B, a hatched portion is an image saving region (the image region thatis used to generate a panorama image), and regions other than that(unsaved regions) are not saved. Also, when the camera is moved in thevertical direction to pick up a panorama image, the relationship betweenthe up-down direction (height direction) and the left-right direction(horizontal direction) is switched, but the same phenomenon occurs. Notethat although only the left side of an image is shown in FIG. 3B, thesame follows for the right side as well.

Also, depending on the transform coefficient in the cylindricalcoordinate transform, there are cases where the up-down direction in animage does not form a straight line. In this case as well, the imagesaving region becomes narrower.

In FIG. 3A, the size reduction that accompanies the cylindricalcoordinate transform is more notable the shorter the focal length is, ascan be understood from a comparison of the case of a focal length (f) of18 mm and the case of 11 mm. Panorama images often have spanninglarge-scale scenery as its shooting subject, and because wide anglelenses that have short focal lengths tend to be used more frequently,the occurrence of the problems mentioned above become more notable.

However, conventional cameras display what is referred to as“through-the-lens images”, which are output from an image sensor to aliquid crystal display (monitor) during the shooting of a panoramaimage. For this reason, a user cannot know which region of athrough-the-lens image will become an image saving region (from whichregions the panorama image will be generated).

SUMMARY OF THE INVENTION

The present invention has been made in view of the above situation, andprovides technology that supports user recognition of image regions thathave a possibility of not being included in a panorama image whenpicking up images that will become the basis of a panorama image.

According to a first aspect of the present invention, there is providedan image pickup apparatus, comprising: an image sensor configured tosuccessively pick up a plurality of images for compositing of a panoramaimage; a memory configured to store instructions; a monitor; and aprocessor connected to the memory and configured to execute theinstructions to: specify a region that is not to be used in compositingof the panorama image in each of the plurality of images, based on astate of an optical system used at a time of pick-up of the plurality ofimages, wherein the monitor displays the specified region along with animage.

According to a second aspect of the present invention, there is provideda control method executed by an image pickup apparatus having an imagesensor and a monitor, comprising: successively picking up, by the imagesensor, a plurality of images for compositing of a panorama image;specifying a region that is not to be used in compositing of thepanorama image in each of the plurality of images, based on a state ofan optical system used at a time of pick-up of the plurality of images;and displaying, on the monitor, the specified region along with animage.

According to a third aspect of the present invention, there is provideda non-transitory computer-readable storage medium which stores a programfor causing a computer of an image pickup apparatus having an imagesensor and a monitor to execute a control method comprising:successively picking up, by the image sensor, a plurality of images forcompositing of a panorama image: specifying a region that is not to beused in compositing of the panorama image in each of the plurality ofimages, based on a state of an optical system used at a time of pick-upof the plurality of images; and displaying, on the monitor, thespecified region along with an image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a digital camera 100.

FIGS. 2A to 2B are diagrams that illustrate a summary of panoramashooting.

FIG. 3A is a conceptual diagram of a cylindrical coordinate transform.

FIG. 3B is a diagram illustrating an image saving region and unsavedregions in an image after a cylindrical coordinate transform.

FIG. 4 is a flowchart of shooting processing that the digital camera 100executes.

FIG. 5 is a flowchart of panorama shooting processing.

FIG. 6 is a diagram showing the data flow of panorama shootingprocessing.

FIG. 7 is a diagram showing an example of a live view display inpanorama shooting processing.

FIG. 8 is a diagram showing an example of a live view display inpanorama shooting processing.

FIG. 9 is a diagram showing an example of a live view display inpanorama shooting processing.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the attached drawings. It should be noted that thetechnical scope of the present invention is defined by the claims, andis not limited by the following respective embodiments. Also, not all ofthe combinations of the aspects that are described in the embodimentsare necessarily essential to the present invention. Also, the aspectsthat are described in the respective embodiments can be combined asappropriate.

First Embodiment

Hereinafter, a summary of panorama shooting will be described withreference to FIGS. 2A and 2B. As shown in FIG. 2A, a plurality of imagesare picked up successively while a user moves a digital camera 100. Themovement of the digital camera 100 may be performed by an automaticcamera platform or the like. As shown in FIG. 2B, the user performsshooting such that common regions (overlapping regions) of a shootingsubject are included in the images that have been picked up.

The digital camera 100 subjects the picked-up images to processing thatcorrects distortion of an optical lens (pick-up lens), and processingfor mapping onto a virtual cylinder that has the focal length of theoptical lens as its radius (cylindrical coordinate transformprocessing). Also, the digital camera 100 extracts feature points in thecommon regions of the images, and detects movement vectors that indicatethe extent of movement of the feature points. The digital camera 100performs a transform into an affine transform coefficient or the likebased on the movement vectors, and overlays two images such that thefeature points are aligned. Accordingly, an image in which portionsother than the common area are expanded is obtained. By repeating this aplurality of times, it is possible to generate a panorama image thatcovers a wider range than a single picked-up image.

FIG. 1 is a block diagram of the digital camera 100. In FIG. 1, 101 is apick-up lens. Reference sign 102 is an AF (autofocus) drive circuit. TheAF drive circuit 102 is constituted by, for example, a DC motor orstepping motor, and performs focusing by changing the focus lensposition of the pick-up lens 101 under control of a microcomputer 123.Reference sign 103 is diaphragm. Reference sign 104 is a diaphragm drivecircuit. The diaphragm drive circuit 104 drives the diaphragm 103. Theamount to be driven is calculated by the microcomputer 123. Thus, theoptical aperture value changes.

Reference sign 105 is a main mirror for the purpose of switching a lightbeam that is incident from the pick-up lens 101 between a viewfinderside and an image sensor 112 side. The main mirror 105 is, under normalconditions, arranged to reflect a light beam such that it is lead to theviewfinder side, but in cases where shooting is performed or the liveview is displayed, it springs upward out of the light beam so as to leadthe light beam to the image sensor 112. Also, the main mirror 105 is ahalf mirror such that the central portion can transmit a portion of thelight, and a portion of the light beam is transmitted so as to beincident on a sensor (not shown) for performing focus detection.

Reference sign 106 is a sub mirror for the purpose of reflecting a lightbeam from the main mirror 105 and leading it to a sensor for performingfocus detection and a sensor that is arranged within an exposure amountcalculation circuit 109. Reference sign 107 is a mirror drive circuitthat drives the main mirror 105 under control of the microcomputer 123.Reference sign 108 is a pentaprism that constitutes a viewfinder. Theviewfinder is also constituted by a focus board (not shown), an eyepiece lens (not shown), and the like.

Reference sign 109 is an exposure amount calculation circuit. The lightbeam that passed through the central portion of the main mirror 105 andwas reflected by the sub mirror 106 arrives at a sensor for performingphotoelectric conversion, which is arranged in the exposure amountcalculation circuit 109.

The defocus amount, which is for use in focus calculations, can beobtained by calculation based on the output of the sensor for performingfocus detection. The microcomputer 123 evaluates the calculation result,instructs the AF drive circuit 102, and drives the focus lens.

Reference sign 110 is a focal-plane shutter. Reference sign 111 is ashutter drive circuit, which drives the focal-plane shutter 110. Theopening time of the focal-plane shutter 110 is controlled by themicrocomputer 123.

Reference sign 112 is an image sensor. The image sensor 112 is a CCD orCMOS sensor or the like, and converts a subject image formed by thepick-up lens 101 into an electrical signal. Reference signal 115 is anA/D converter. The A/D converter 115 converts an analog output signal ofthe image sensor 112 into a digital signal.

Reference sign 116 is a video signal processing circuit and is realizedby a logic device such as a gate array. The video signal processingcircuit 116 includes a luminance adjustment circuit 116 a that adjuststhe brightness according to a digital gain, a gamma correction circuit116 b that adjusts luminance according to a gamma characteristic, and amovement amount calculation circuit 116 c that calculates an amount ofshake (movement amount) of a plurality of images. The video signalprocessing circuit 116 also includes an alignment circuit 116 d thatperforms alignment according to the movement amount, a geometrictransform circuit 116 e that corrects distortion of the pick-up lens 101and performs a cylindrical coordinate transform, and a scaling circuit116 f that changes the size of an image. The video signal processingcircuit 116 also includes a trimming circuit 116 g that extracts partsof an image, a compositing circuit 116 j that composites a plurality ofimages, a developing circuit 116 k, a compressing/expanding circuit 116l that performs processing for conversion into a standard image formatsuch as JPEG, and the like.

Reference sign 117 is a display drive circuit. Reference sign 118 is adisplay apparatus using TFT, organic EL, or the like. Reference sign 119is a memory controller. Reference sign 120 is a memory. Reference sign121 is an external interface that can connect to a computer and thelike. Reference sign 122 is a buffer memory.

The video signal processing circuit 116 performs filter processing,color conversion processing, and gamma processing on digitized imagedata, along with JPEG compression processing or the like, and outputs itto the memory controller 119. At this time, it is possible totemporarily put an image in the buffer memory 122 as it is beingprocessed. The video signal processing circuit 116 can also output avideo signal from the image sensor 112 or output image data that hasbeen conversely input from the memory controller 119 to the displayapparatus 118 via the display drive circuit 117. The switching of thesefunctions is performed according to instructions from the microcomputer123.

The video signal processing circuit 116 can output information such aswhite balance or exposure information of a signal from the image sensor112 as needed to the microcomputer 123. The microcomputer 123 giveswhite balance or gain adjustment instructions based on this information.The microcomputer 123 can temporarily store picked-up data in the buffermemory 122 as unprocessed images in the case of a continuous shootingoperation. Afterwards, the microcomputer 123 reads out the unprocessedimage data through the memory controller 119, performs image processingand compression processing with the use of the video signal processingcircuit 116, and performs continuous shooting. The number of imagestaken continuously is determined by the size of the buffer memory 122 orby the size of the image in the case of panorama shooting.

The memory controller 119 stores unprocessed digital image data inputfrom the video signal processing circuit 116 in the buffer memory 122,and stores processed digital image data in the memory 120. Also, thememory controller 119 conversely outputs image data from the buffermemory 122 and the memory 120 to the video signal processing circuit116.

The memory 120 can, in some instances, be removed. The memory controller119 can output an image that is stored in the memory 120 via theexternal interface 121 that can connect to a computer and the like.

Reference sign 123 is a microcomputer. Reference sign 124 is anoperating member. The operating member 124 conveys its status to themicrocomputer 123, and the microcomputer 123 controls the units of thedigital camera 100 according to changes in the status of the operatingmember 124.

Reference sign 125 is a switch 1 (SW1). Reference sign 126 is a switch 2(SW2). The SW1 125 and SW2 126 are switches that that turn on and offaccording to operations of a release button, and are each one of theinput switches of the operating member 124. The state where only the SW1125 is on is the state where the release button is pressed halfway, andin this state, the microcomputer 123 performs a photometric operationalong with an autofocus operation. The state where the SW1 125 and theSW2 126 are both on is the state where the release button is fullypressed, which is also the ON state of the release button for recordingan image. Image pick-up is performed in this state. Also, a continuousshooting operation is performed while the SW1 125 and the SW2 126 arecontinually in the ON state.

The operating member 124 is also connected to switches (not shown) thatcorrespond to an ISO setting button, a menu button, a set button, aflash setting button, and a single shot/continuous shot/self-timerbutton. The operating member 124 is also connected to switches (notshown) that correspond to a move + (plus) button and a move − (minus)button for moving through menus and playback images, an exposurecorrection button, a display image enlargement button, a display imagereduction button, and a playback switch. The operating member 124 isalso connected to switches that correspond to an aperture button thatchanges the diaphragm 103 to the set aperture, an erase button forerasing images that have been picked-up, an information display buttonrelated to shooting and playback, and the like. The microcomputer 123detects the status of each switch. Also, it is possible to more easilyselect functions and values by providing a rotary dial switch to realizethe functions of the plus button and the minus button.

Reference sign 127 is a liquid crystal drive circuit. Reference sign 128is an external liquid crystal display apparatus. Reference sign 129 isan in-finder liquid crystal display apparatus. In accordance withdisplay content instructions from the microcomputer 123, the liquidcrystal drive circuit 127 drives the external liquid crystal displayapparatus 128 and the in-finder liquid crystal display apparatus 129that display an operational status, a message, and the like, usingletters and images. Also, the in-finder liquid crystal display 129 isprovided with back lighting such as an LED (not shown), and the LED isalso driven by the liquid crystal drive circuit 127.

The microcomputer 123 can calculate the remaining number of images thatcan be picked up based on a check of the memory capacity through thememory controller 119, based on predictive value data of an image sizeaccording to ISO sensitivity, image size, and image quality, which areset before pick-up. The microcomputer 123 can display the remainingnumber of possible images on the external liquid crystal displayapparatus 128 and the in-finder liquid crystal display apparatus 129 asneeded.

Reference sign 130 is a nonvolatile memory (EEPROM) that can save dataeven when the digital camera 100 is not powered on. Reference sign 131is a power source unit. The power source unit 131 supplies powernecessary for the IC's and drive systems. Reference sign 132 is a clock.The microcomputer 123 uses the clock unit 132 to save the time ofpick-up to an image file recorded to the memory 120 and superimposes thetime of pick-up onto an image.

Reference sign 133 is a gyroscope. The gyroscope 133 detects the angularvelocity of rotation of the digital camera 100 in two axes or threeaxes.

Reference sign 134 is a thermometer. The thermometer 134 is preferablyarranged in the vicinity of the gyroscope sensor, but normally may be asensor arranged in the vicinity of the image sensor 112, the memory 120or the like for the protection of the digital camera 100 or the memory120.

The shooting processing executed by the digital camera 100 will bedescribed below with reference to FIG. 4. Unless otherwise noted, theprocessing of the steps in the flowchart in FIG. 4 is achieved by themicrocomputer 123 controlling the units of the digital camera 100 byexecuting control programs stored in the nonvolatile memory 130. Theshooting processing begins when the SW2 126 is pressed down by the user.Note that the microcomputer 123 calculates the exposure amount via theexposure amount calculation circuit 109, and sets the aperture,accumulation time, and ISO sensitivity in advance.

In step S402, the microcomputer 123 performs exposure. Specifically, themicrocomputer 123 notifies the aperture determined in advanced to thediaphragm drive circuit 104 and sets the diaphragm 103 to the targetaperture. Also, the microcomputer 123 powers on the image sensor 112,the A/D converter 115, and the like, and performs preparation forshooting. When the preparation is complete, the microcomputer 123 drivesthe mirror drive circuit 107 and causes the subject image to be formedon the image sensor 112. The shutter drive circuit 111 opens a frontcover (not shown) of the focal-plane shutter 110 and causes a shootingsubject image to form on the image sensor 112. Then, the shutter drivecircuit 111 closes a back cover (not shown) of the focal-plane shutter110 after a predetermined accumulation time, and causes light to enterinto the image sensor 112 only during the pre-determined accumulationtime.

In step S403, the microcomputer 123 reads out an image signal to thevideo signal processing circuit 116 via the A/D converter 115 and storesthe image signal in the buffer memory 122.

In step S404, microcomputer 123 uses the developing circuit 116 k andperforms development that converts the read-out image signal into imagedata. At this time, the microcomputer 123 can perform image processingsuch as white balance processing to obtain an appropriate image, orprocessing which uses the gamma correction circuit 116 b to apply gainto dark portions.

In step S405, the microcomputer 123 uses the compressing/expandingcircuit 116 l to perform compression processing that converts theobtained image data to a general purpose data format such as JPEG.

In step S406, the microcomputer 123 saves the compressed image data tothe memory 120, such as an SD card or a compact flash drive (registeredtrademark).

Note that a configuration is possible in which the microcomputer 123,does not performing the image processing and development processing ofstep S404, and instead losslessly compresses the read-out image signalin step S405 as it is, and saves the losslessly compressed image signalto the memory 120 in step S406. The switch of whether or not to performcompression processing can be made by the user by using the operatingmember 124.

The following describes a summary of a panorama shooting mode. Panoramashooting has a mode in which the digital camera 100 is moved in ahorizontal direction during shooting and a mode in which it is moved ina vertical direction during shooting, but the following shows an exampleof performing shooting while moving in the horizontal direction.

If the user sets the digital camera 100 to the panorama shooting modevia the operating member 124, the microcomputer 123 supplies power tothe image sensor 112 and the A/D converter 115, and performs initialsetting. Also, the microcomputer 123 raises the main mirror 105, and theshutter drive circuit 111 opens the focal-plane shutter 110 to form asubject image on the image sensor 112 through the pick-up lens 101.

The signal from the image sensor 112 is converted into a digital signalby the A/D converter 115 and is developed by the developing circuit 116k of the video signal processing circuit 116. Also, the developed signalis converted into an appropriate image by the luminance adjustmentcircuit 116 a and the gamma correction circuit 116 b, and is scaled toan appropriate size for the display apparatus 118 by the scaling circuit116 f, and is displayed on the display apparatus 118. By repeating this24 to 60 times per second, a so-called live view can be displayed.

The user sets the angle of view while checking the live view display. Inthe case of panorama shooting, the user faces the digital camera 100toward a subject that will become a main subject in a wide range andpresses the SW1 125. The microcomputer 123 calculates an exposure amountwhen the SW1 125 is pressed. If the live view is not being displayed,the exposure amount calculation circuit 109 receives light that has beenreflected by the sub-mirror 106 and calculates an optimum exposureamount. While the live view is being displayed, an exposure amountcalculation circuit (not shown) included in the video signal processingcircuit 116 calculates an optimum exposure amount. The microcomputer 123uses the diaphragm drive circuit 104 to drive the diaphragm 103 based onthe calculated exposure amount, and controls the accumulation time andthe sensitivity of the image sensor 112. Also, the AF drive circuit 102drives the pick-up lens 101 and obtains focus. When preparation forshooting is complete, the microcomputer 123 notifies the user via abuzzer (not shown) or the like.

Next, the user faces the digital camera 100 in an initial desiredshooting direction and then presses the SW2 126 down. The digital camera100 performs the actual panorama shooting when the SW2 126 is pressed.

Details of panorama shooting will be described below with reference toFIGS. 5 and 6. Unless otherwise noted, the processing of the steps ofthe flowchart of FIG. 5 is achieved by the microcomputer 123 controllingthe units of the digital camera 100 by executing control programs storedin the nonvolatile memory 130.

In step S500, the microcomputer 123 acquires lens information. This lensinformation includes data for the correction of distortion that isdescribed later and low light levels of a lens peripheral portion, thefocal distance used in a cylindrical coordinate transform also describedlater, and the like.

In step S501, the microcomputer 123 picks up the first image. Becausethe image sensor 112 and the A/D converter 115 are set to driving forlive view, the microcomputer 123 switches to driving for still imageshooting. The microcomputer 123 adjusts the diaphragm 103 to thepreviously determined exposure amount, opens and shuts the focal-planeshutter 110, and causes a subject image to form on the image sensor 112.As described above, the subject image that forms on the image sensor 112is converted into a digital signal by the A/D converter 115, and isstored in the buffer memory 122. Correction of shading of the imagesensor 112 and the like is performed on image data obtained in thismanner via a circuit (not shown) included in the video signal processingcircuit 116. Such image data that has undergone a minimal level ofprocessing is called a RAW image. As shown in FIG. 6, a RAW image 605 isdeveloped by the developing circuit 116 k and becomes a developed image(YUV image) 606. In order to display the picked-up image on the displayapparatus 118, the scaling circuit 116 f reduces the developed image 606according to the pixel count of the display apparatus 118, and stores itin a VRAM 608.

In step S502, the microcomputer 123 initializes (resets) the gyroscope133. This processing is performed in order to acquire the extent thatthe digital camera 100 has been swung (rotated) up to the point when thesecond image is to be picked up in step S506.

In step S503, the microcomputer 123 performs distortion correctionprocessing. That is, as shown in FIG. 6, processing that correctsdistortion aberration of the pick-up lens 101 is performed on thedeveloped image 606 by the geometric transform circuit 116 e usingexisting technology.

In step S504, the microcomputer 123 calculates the radius of a virtualcylinder in order to perform a cylindrical coordinate transform, basedon the focal distance acquired in step S502.

In step S505, the microcomputer 123 uses the radius of the virtualcylinder that was calculated in step S504 to perform a cylindricalcoordinate transform. As shown in FIG. 6, the geometric transformcircuit 116 e performs a cylindrical coordinate transform on thedeveloped image 606. Also, the microcomputer 123 determines and extractsan image saving region (FIG. 3B) in the post-cylindrical coordinatetransform image. Thus, a post-geometric transform image 607 isgenerated. Note that the determination criteria for the image savingregion is not particularly limited, but, for example, the microcomputer123 determines an image saving region such that an adequate image savingheight can be obtained. As shown in FIG. 3A, the pixel count in theheight direction at the same horizontal position in the post-cylindricalcoordinate transformed image decreases the shorter the focal distanceis. Accordingly, is it possible to acquire an adequate saved imageheight regardless of the focal distance by determining an image savingregion such that the width of the image saving region decreases theshorter the focal distance is.

Note that because both the cylindrical coordinate transform andaberration distortion correction are performed by the geometrictransform circuit 116 e, they can be performed as separate steps asdescribed above, or can be performed simultaneously.

In step S506, the microcomputer 123 picks up the second image. Thedetails of the shooting are the same as in step S501. Note that in FIG.6, the post-geometric transform image 607 that corresponds to theshooting of the first image becomes the post-geometric transform image603 during the shooting of the second image.

In step S507, the microcomputer 123 acquires gyroscope information inorder to obtain the amount of swing from the previous shooting. Themicrocomputer 123 acquires gyroscope information with respect to twoaxes, namely the yaw direction and the pitch direction relative to thedigital camera 100, but it can acquire three axes, further including theroll direction that rotation about the optical axis. The output from thegyroscope 133 is angular velocity. However, because information relatedto the extent of swing from the previous shooting in panorama shootingis necessary, the microcomputer 123 integrates the angular velocity fromthe previous shooting until the present shooting, and calculates therotational angle from the previous shooting when shooting the second andsubsequent images. The microcomputer 123 stores the calculatedinformation as gyroscope information 604 shown in FIG. 6. Then, themicrocomputer 123 resets the gyroscope 133 in the same manner as in stepS502.

In step S508, the microcomputer 123 converts the rotational angle intothe movement amount of pixel units based on the focal distance and theangle of view of the lens obtained in step S502, the image sensor 112information, and the like. In general, the angle of view (α) of adistortionless lens or after distortion correction is expressed below inExpression 1, where the effective focal distance is f [mm] and the widthof the image sensor 112 is w [mm].α[°]=2×arctan(w[mm]÷2÷f[mm])  (1)

Letting the size of one pixel on the image sensor 112 be p [μm] and theswing angle [°] be θ, a movement amount d [pix] in an image is expressedby Expression 2 below.d[pix]=tan(α[°]÷2)×f[mm]p[μm]×1000  (2)

In step S509, the microcomputer 123 performs distortion aberrationcorrection on the second image in the same manner as the distortionaberration correction for the first image in step S505.

In step S511, in the same manner as step S505, the microcomputer 123uses the geometric transform circuit 116 e to perform a cylindricalcoordinate transform and extraction of the image saving region, andgenerates the post-geometric transform image 607 shown in FIG. 6.

In step S512, the microcomputer uses the movement amount calculationcircuit 116 c to calculate the movement amount based on the previouspick-up image and the current pick-up image. As shown in FIG. 6, themovement amount calculation circuit 116 c calculates the amount ofmovement between the current image (the post-geometric transform image607) and the previous image (the post-geometric transform image 603).Known methods can be used in relation to the calculation of the movementamount, but in the present embodiment, the movement amount calculationcircuit 116 c finds several feature points in an image and calculates anaffine coefficient 609 by sampling. The movement amount calculationcircuit 116 c detects an edge, extracts a feature point, and thencalculates the movement amount. For example, assume that a feature point1 has shifted from coordinates (x1, y1) to coordinates (u1, v1), afeature point 2 has shifted from coordinates (x2, y2) to coordinates(u2, v2), and a feature point 3 has shifted from coordinates (x3, y3) to(u3, v3). In this case, Expressions 3 and 4 are obtained whenrepresented as a simultaneous equation.

$\begin{matrix}{{\begin{pmatrix}{x\; 1} & {y\; 1} & 1 \\{x\; 2} & {y\; 2} & 1 \\{x\; 3} & {y\; 3} & 1\end{pmatrix}\begin{pmatrix}a \\b \\c\end{pmatrix}} = \begin{pmatrix}{u\; 1} \\{u\; 2} \\{u\; 3}\end{pmatrix}} & (3) \\{{\begin{pmatrix}{x\; 1} & {y\; 1} & 1 \\{x\; 2} & {y\; 2} & 1 \\{x\; 3} & {y\; 3} & 1\end{pmatrix}\begin{pmatrix}d \\e \\f\end{pmatrix}} = \begin{pmatrix}{v\; 1} \\{v\; 2} \\{v\; 3}\end{pmatrix}} & (4)\end{matrix}$

By solving these equations, it is possible to calculate affinecoefficients a to f. If four or more feature points are detected, theclosest points are excluded, and the “least-squares” method is used tonormalize. If three points are not found, if three extracted points areon a straight line, or if two of three points are close, those valuesare examined, and it is determined that the movement amount calculationfailed.

If there is a large difference between the movement amount calculatedfrom an image in this way (affine coefficient 609) and the movementamount calculated in step S508 (movement amount based on the gyroscopeinformation 604), it is conceivable the that a repeated pattern or amoving body is included in the image. Therefore, the microcomputer 123may calculate the movement amount again under different conditions, maytreat this shot as a failure and return to the next shooting (S506), ormay handle it as failed panorama shooting.

In step S513, the microcomputer 123 uses the alignment circuit 116 d toperform alignment based on the movement amount (affine coefficient)calculated in step S512. That is, as shown in FIG. 6, the alignmentcircuit 116 d uses the affine coefficient 609 to perform alignment, andgenerates an aligned image 611.

In step S514, the microcomputer 123 uses the compositing circuit 116 jto composite images. As shown in FIG. 6, if the current pick-up image isthe (N−1)-th image, the compositing circuit 116 j composites a compositeimage 610 that is based on the first to the (N−1)-th images and thealigned image 611 that is based on the N-th image. Thus, a compositeimage 612 is generated. Also, if N=2, the composite image 610 will beequivalent to the post-geometric transform image 603. Also, because theimage quality of the composite result will decrease in the case of amoving body such as the surface of water, the compositing circuit 116 jmay improve the image quality by changing the composite ratio in theboundary portions of images.

In step S515, the microcomputer 123 determines whether or not an endinstruction has been given. If an end instruction has been given,processing proceeds to step S516. If an end instruction has not beengiven, processing returns to step S506, and if the switch SW2 126 ispressed again, the microcomputer 123 performs the next shooting, andprocessing is executed from step S507 to step S514 in the same manner asthe previous shooting. Accordingly, using a plurality of partial images(image saving regions) that have been obtained from a plurality ofimages successively shot, the microcomputer 123 generates a panoramaimage by performing image compositing to align the overlapping regionsof each pair of two partial images that are sequential in the shootingorder.

In step S516, the microcomputer 123 uses the compressing/expandingcircuit 116 l to compress the composite image 612 into a general purposeformat such as JPEG.

In step S517, the microcomputer 123 saves the compressed image to thememory 120. Also, at this time, the gamma correction circuit 116 b mayperform γ correction so that dark parts of the composite image are madeeasier to view, and perform corrections in order to unify the color toneof the overall image. Also, because the image obtained in this way has alarge size, the scaling circuit 116 f may perform scaling of the imageto a size instructed by the user in advance. Furthermore, inconsideration of hand shaking or the like, the trimming circuit 116 gmay perform image clipping by the largest inscribed rectangle, or by apredetermined region, before the image is saved.

Note that this shows an example of the digital camera 100 being swung inthe horizontal direction, but similar processing can be executed in thecase of being swung in the vertical direction.

The following describes, with reference to FIG. 7, the live view displaythat is performed during shooting standby in the panorama shootingprocessing of FIG. 5. Reference sign 700 in FIG. 7 is a through-the-lensimage. Reference signs 701, 702, 703 are the boundary lines between theimage saving regions and other regions (unsaved regions). Reference sign704 shows the direction of movement of the digital camera 100 duringpanorama shooting.

It is desirable that the display position of the border line 701 is, inthe image after a cylindrical coordinate transform has been performedbased on the focal length, the end of a region that is not saved as apanorama image (unsaved region). Also, it is desirable for the user tobe able to recognize the unsaved region on the side opposite to theshooting direction in the shooting standby mode before the start ofpanorama image shooting. Also, the display positions of border lines 702and 703 are, in the image after a cylindrical coordinate transform hasbeen performed based on the focal length, desirably at the end in theheight direction of regions that are not saved as a panorama image.Also, it is desirable to appropriately perform region display by adding,to the cylindrical transform amount, the shake amount in the case wherethe shooting direction of the digital camera 100 is moved.

Accordingly, in FIG. 7, the hatched regions correspond to parts ofunsaved regions, while a white region corresponds approximately to animage saving region. As described in step S505 of FIG. 5, a reduction inthe number of pixels in the height direction in the right side ofportion of an image will occur when a cylindrical coordinate transformis performed, and therefore this portion is not extracted as an imagesaving region. Accordingly, the regions used in composite processing instep S514 are, among the white regions, regions that do not include theportions on the right side. However, in panorama shooting that swingsthe digital camera 100 in the direction of an arrow 704, there is apossibility that the right side part will be included in the imagesaving region in the subsequent shooting, and therefore is possiblyincluded in the final panorama image.

Also, the microcomputer 123 may display the unsaved regions of the rightside of the through-the-lens image so they are distinguishable from theimage saving region. Accordingly, in the present embodiment, themicrocomputer 123 may display parts of the unsaved regions so that theuser can recognize them, or may display the unsaved regions in theirentirety so that the user can recognize them. In other words, whenperforming live viewing, the microcomputer 123 displays information thatindicates at least a portion of unsaved regions (predetermined regions)of the subsequent pickup image.

Also, in panorama shooting, a plurality of images (unit images) arecomposited to create a panorama image. For this reason, when startingshooting, it is preferable to display the end on side opposite to themovement direction of the digital camera 100 (at least a portion of theregion positioned at the side opposite to the movement direction of thedigital camera 100). However, as shown in FIG. 8, it is preferable todisplay the end on the same side as the movement direction of thedigital camera 100 during shooting.

In FIG. 8, 800 is a through-the-lens image. Reference signs 801, 802 and803 are the border lines between image saving regions and regions otherthan those (unsaved regions). Reference sign 804 shows the direction inwhich the digital camera 100 moves during panorama shooting. In thisway, it is possible for the user to accurately know to what extent animage has been saved by the accurate display of the end on the same sideas the movement direction of the digital camera 100 (at least a portionof the region positioned on the same side as the movement direction ofthe digital camera 100).

Additionally, the digital camera 100 may be configured such that it canswitch between the display mode of FIG. 7 and the display mode of FIG. 8at an appropriate timing (a timing when a predetermined condition ismet). In general, during panorama shooting, the lowest pixel count for asaved image is often set. Therefore, the digital camera 100 employs thedisplay mode of FIG. 7 that displays the end on the side opposite to themovement direction of the digital camera 100 at the start of shooting.Also, if the movement amount of the digital camera 100 reaches amovement amount that corresponds to the minimum pixel count of a savingimage, the digital camera 100 switches from the display mode of FIG. 7to the display mode of FIG. 8. In other words, if the pixel count of apanorama image that can be generated from image saving regions (partialimages) corresponding to one or more images already picked-up and thenext image to be picked up, is greater than or equal to a thresholdvalue, a switch from FIG. 7 to FIG. 8 is performed. Alternatively, thedigital camera 100 may perform a switch from FIG. 7 to FIG. 8 based onthe number of picked-up images. For example, the digital camera 100performs a switch from FIG. 7 to FIG. 8 if the number of images thathave already been picked up is greater than or equal to a thresholdvalue.

Also, as shown in FIG. 9, the digital camera 100 may, in synchronizationwith shooting, successively reduce the size of the region that isdivided by the border line 701 of FIG. 7 according to the movementamount. That is, in FIG. 9, the digital camera 100 displays at least aportion of a region that is not included in the eventually obtainedpanorama so that it can be recognized by the user. In other words, thedigital camera 100 displays information that indicates at least aportion of, in an unsaved region (predetermined region) of an image tobe picked up next, a region included in each unsaved region of one ormore images that have already been picked up.

Also, in the same manner, if the movement amount of the digital camera100 reaches a movement amount that corresponds to the minimum pixelcount of saved images, the digital camera 100 may successively enlargethe region that is divided by the border line 801 of FIG. 8 from the endon the side of the movement direction of the digital camera 100 to apredetermined region. The predetermined region mentioned here is assumedto be a region that has been set based on the cylindrical coordinatetransform and the like. Also, displays may be switched in a directionperpendicular to the movement direction of the digital camera 100. Thatis, the regions divided by the borderline 701 or the regions divided bythe borderline 801 may be switched.

Also, in FIGS. 7 to 9, the unsaved regions are indicated with hatching,but the display mode of the present embodiment is not limited to this,and it is possible to employ any display mode that allow the user torecognize unsaved regions.

As described above, according to the first embodiment, the digitalcamera 100 performs live view display on the display apparatus 118during successive shooting of a plurality of images at the same time asbeing moved in the direction of shooting. At this time, the digitalcamera 100 displays information that indicates at least a portion of anunsaved region (predetermined region) of an image that is to be pickedup subsequently. Thus, it is possible to support user recognition ofimage regions that will potentially not be included in a panorama imageduring shooting of images that will become the base for a panoramaimage.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-149313, filed Aug. 1, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image pickup apparatus, comprising: an imagesensor configured to successively pick up a plurality of images forcompositing of a panorama image; a memory configured to storeinstructions; a monitor; and a processor connected to the memory andconfigured to execute the instructions to: specify a region that is notto be used in compositing of the panorama image in each of the pluralityof images, based on a movement state of an optical system used at a timeof pick-up of the plurality of images, wherein the monitor displays thespecified region along with an image.
 2. The image pickup apparatusaccording to claim 1, wherein the processor is further configured toexecute the instructions to: apply a geometric transform to theplurality of images, based on the movement state of the optical systemused at the time of pick-up of the plurality of images, extract apartial region for use in compositing of the panorama image, from theimages to which the geometric transform was applied, and specify aregion that corresponds to a region other than the partial region as theregion that is not to be used in compositing of the panorama image. 3.The image pickup apparatus according to claim 2, wherein the geometrictransform includes at least one of distortion aberration correction anda cylindrical coordinate transform.
 4. The image pickup apparatusaccording to claim 1, wherein the processor is further configured toexecute the instructions to generate the panorama image by performingimage compositing on the plurality of images to superimpose overlappingregions of each pair of two images that are sequential in a pickuporder.
 5. The image pickup apparatus according to claim 1, wherein themonitor displays the specified region at a position on a side oppositeto a movement direction in a pick-up direction in the image.
 6. Theimage pickup apparatus according to claim 1, wherein before apredetermined condition is met, the monitor displays the specifiedregion at a position on a side opposite to a movement direction in apick-up direction in the image, and after the predetermined condition ismet, the monitor displays the specified region at a position on a sameside as the movement direction in the pick-up direction in the image. 7.The image pickup apparatus according to claim 6, wherein thepredetermined condition is met in a case where a pixel count of thepanorama image that can be generated from one or more previouslypicked-up images and an image to be picked-up subsequently is greaterthan or equal to a first threshold value.
 8. The image pickup apparatusaccording to claim 6, wherein the predetermined condition is met in acase where a number of previously picked-up images is greater than orequal to a second threshold value.
 9. The image pickup apparatusaccording to claim 1, wherein the monitor displays information thatindicates at least a portion of, in the specified region of an image tobe picked-up, a region included in each specified region of one or morepreviously picked-up images.
 10. A control method executed by an imagepickup apparatus having an image sensor and a monitor, comprising:successively picking up, by the image sensor, a plurality of images forcompositing of a panorama image; specifying a region that is not to beused in compositing of the panorama image in each of the plurality ofimages, based on a movement state of an optical system used at a time ofpick-up of the plurality of images; and displaying, on the monitor, thespecified region along with an image.
 11. A non-transitorycomputer-readable storage medium which stores a program for causing acomputer of an image pickup apparatus having an image sensor and amonitor to execute a control method comprising: successively picking up,by the image sensor, a plurality of images for compositing of a panoramaimage; specifying a region that is not to be used in compositing of thepanorama image in each of the plurality of images, based on a movementstate of an optical system used at a time of pick-up of the plurality ofimages; and displaying, on the monitor, the specified region along withan image.
 12. An image picjup apparatus, comprising: an image sensorconfigured to successively pick up a plurality of images for compositingof panorama image; a memory configured to store instructions; a monitorconfigured to display a region in each of the plurality of images; and aprocessor connected to the memory and configured to execute theinstructions to: change the region when a movement state of an opticalsystem used at a time of pick-up of the plurality of images changes. 13.The image pickup apparatus according to claim 12, wherein the processorfurther executes the instructions to change a size of the region whenthe movement state changes.
 14. The image pickup apparatus according toclaim 12, wherein the region is to be used in compositing of thepanorama image in each of the plurality of images.
 15. A control methodexecuted by an image pickup apparatus having an image sensor and amonitor, comprising: successively picking up, by the image sensor, aplurality of images for compositing of a panorama image; changing aregion when a movement state of an optical system used at a time ofpick-up of the plurality of image changes; and displaying, on themonitor, the region in each of the plurality of images.
 16. Anon-transitory computer-readable storage medium which stores a programfor causing a computer of an image pickup apparatus having an imagesensor and a monitor to execute a control method comprising:successively picking up, by the image sensor, a plurality of images usedfor compositing of a panorama image; changing a region when a movementstate of an optical system used at a time of pick-up of the plurality ofimages changes; and displaying, on the monitor, the region in each ofthe plurality of images.